Abstract

The Station/Meridian team is aiming for major product releases at LabWeek 2023 (Nov 13-17). We likely won’t be able to get everything done as requested by stakeholders, so we need to negotiate a configuration of deliverables that works.

• @Julian Gruber: adjust dependencies and options now that we have 4 deliverables, and other Spark/Meridian requirements

Deliverables

The key deliverables are:

• STA Station public release with rewards

• MER Meridian Impact Evaluator framework
  • Potential variables:
    • Smart contract tasking (instead of centralized tasking service)
    • Smart task sampling (instead of basic random sampling)
    • Smart contract membership management (instead of implicit membership)

• SPA Spark running on Meridian

• RAT Retrieval attestation
  • ⚠️ There probably isn’t going to be a Boost release with this builtin until labweek. Maybe we won’t even be able to ship a special tagged version towards participating SPs

Dependencies

• STA hard depends on RAT, as without fraud detection neither the rewards paid nor the data produced will be right

• STA soft depends on MER, as while it can’t pay rewards without it, it can pay rewards with a partially implemented Meridian - for example one with a centralized instead of a smart contract orchestrator

• MER soft depends on STA and transitively RAT, as Station/Spark will be the living implementation to ship together with the spec

• RAT soft depends on STA and transitively MER, as while attestation can still be implemented, the network of data producers will be small without rewards

Options

Prioritise Station

➕ STA

➖ MER

➕ RAT

This is the current plan laid out in . We ship STA with RAT but don’t properly release MER . The IE smart contract will be missing , which is a critical part of Meridian.

Prioritise Meridian

➖ STA

➕ MER

➖ RAT

This releases MER with an IE smart contract, which runs the epoch loop and performs tasking. There will be a living implementation in Station, but without rewards, as we wont implement RAT. Since there are no rewards, we won’t be able to ship STA.

IE smart contract without tasking

➕ STA

❔ MER

➕ RAT

It is not clear to me whether tasking is part of the measure step. If it doesn’t have to be, we can ship a more light-weight IE smart contract, and thereby also more likely ship Station with rewards.

IE smart contract without explicit membership management

➕ STA

❔ MER

➕ RAT

It is not clear to me whether membership management has to be part of the IE. Actors in the system can be deducted from the records (measures) submitted to the IE. If it doesn’t have to be, we can ship a more light-weight IE smart contract, and thereby also more likely ship Station with rewards.

Prioritise Retrieval Attestation

➖ STA

➖ MER

➕ RAT

This is arguably the worst possible outcome, as while the data will be more correct, there is no large network of Stations performing the retrievals, and no rewards for Station operators. Retrieval attestation on its own also has the least stakeholders.

Prioritise everything

aka Everything Everywhere All At Once

➕ STA

➕ MER
➕ RAT

➖ TEAM

While this is possible, it is the most risky route, with the most stress and largest potential for failure. How can we simplify all 3 deliverables enough? This clearly is the most desirable outcome that satisfies all the stakeholders.

Ship with testnet FIL

Could this create enough interest to download Stations?

➕ STA

➕ MER
➖ RAT

Options for job scheduling

Tasks & Milestones to consider

• Most likely, we will need to rework how we are scheduling jobs. For example, we will need multiple clients to perform the same (cid, address) check so that we can compare the attestations reported. We may need more changes, e.g. replace the hard-coded list of CIDs with a random walk of IPNI or Filecoin deals.
  • See
  • Subtask: Model what size of committee of Station nodes we need to get honest majority.
  • We can start with 10 nodes to us get started.
  • For each epoch, we want each /24 block of IPv4 address to get at most one job assigned.

• Based on the latest progress on SPARK Fraud Detection design, the milestone Meridian: Boost retrieval attestation is only one part of what we want to implement. The other parts are building Honest Majority to have trust in (data length, blake3 hash) data submitted by SPARK checkers and verifying inclusion proofs submitted by SPARK checkers.

• We need IPNI to implement a new HTTP API endpoint allowing SPARK Orchestrator to get a random CID, see

• IPIP to specify new CAR metadata block returned by Boost - https://github.com/ipfs/specs/pull/431

• Boost to send CAR metadata block when requested by the client - https://github.com/filecoin-project/boost/issues/1610

• Lassie must understand this extra block and forward it in the response

• Frisbee to send CAR metadata block when requested by the client

• Eventually, we should implement GW conformance tests for IPIP-0431

Julian’s hot take

We will ship Meridian, with a marketing site, a contract spec, and a working example using Station and simplified Spark.

We will have a talk “Our road towards Retrieval Attestation”, outlining where we are, the work done, and where we need to go.

We won’t ship Station (mainnet rewards), as retrieval attestation and a good-enough Spark Meridian implementation for rewards is more than our timeframe allows.